Improving question answering performance using knowledge distillation and active learning

Contemporary question answering (QA) systems, including Transformer-based architectures, suffer from increasing computational and model complexity which render them inefficient for real-world applications with limited resources. Furthermore, training or even fine-tuning such models requires a vast amount of labeled data is often not available the task at hand. In this manuscript, we conduct comprehensive analysis mentioned challenges introduce suitable countermeasures. We propose novel knowledge distillation (KD) approach to reduce parameter pre-trained bidirectional encoder representations transformer (BERT) system utilize multiple active learning (AL) strategies immense reduction in annotation efforts. show efficacy our by comparing it four state-of-the-art (SOTA) Transformers-based namely KroneckerBERT, EfficientBERT, TinyBERT, DistilBERT. Specifically, outperform KroneckerBERT21 EfficientBERTTINY 4.5 0.4 percentage points EM, despite having 75.0% 86.2% fewer parameters, respectively. Additionally, achieves comparable performance 6-layer TinyBERT DistilBERT while using only 2% their total trainable parameters. Besides, integration AL approaches into BERT framework, that SOTA results on QA datasets can be achieved when use 40% data. Overall, all demonstrate effectiveness achieving performance, extremely reducing number parameters labeling Finally, make code publicly https://github.com/mirbostani/QA-KD-AL.